Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[WIP] Apply patches from Patchwork payload #295

Closed
wants to merge 1 commit into from
Closed

[WIP] Apply patches from Patchwork payload #295

wants to merge 1 commit into from

Conversation

yurinnick
Copy link
Contributor

@yurinnick yurinnick commented Jul 31, 2023 •
edited by gctucker
Loading

Fixes kernelci/kernelci-api#307
Depends on kernelci/kernelci-core#2041

Looks up for Patchwork payload in node.data and apply patches by mbox url before packaging tree into a tarball.
We expect following minimal structure for Patchwork payload:

{
    "version": 1,
    "patches": [
      {
        "id": int,
        "hash": string,
        "web_url": string,
        "date": datetime,
        "mbox": string
      }
    ]
}

Test run logs:

kernelci-pipeline-notifier            | 07/31/2023 11:33:19 PM UTC [INFO] 2023-07-31 23:33:19.800845  HEAD          64c844bfe8ad32503e18af61  Running    -         checkout
kernelci-pipeline-tarball             | 07/31/2023 11:33:19 PM UTC [INFO] Updating repo for mainline
kernelci-pipeline-tarball             | From https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux
kernelci-pipeline-tarball             |  * branch                      HEAD       -> FETCH_HEAD
kernelci-pipeline-tarball             | From git://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux
kernelci-pipeline-tarball             |  * branch                      HEAD       -> FETCH_HEAD
kernelci-pipeline-tarball             | HEAD is now at 5d0c230f1de8 Linux 6.5-rc4
kernelci-pipeline-tarball             | 07/31/2023 11:33:27 PM UTC [INFO] Repo updated
kernelci-pipeline-tarball             | 07/31/2023 11:33:27 PM UTC [INFO] Applying patch to the repo, patch mbox url: https://patchwork.kernel.org/project/linux-hardening/patch/20230726231139.never.601-kees@kernel
.org/mbox/
kernelci-pipeline-tarball             | 07/31/2023 11:33:28 PM UTC [INFO] Making tarball linux-mainline-master-v6.5-rc4-1-g66f6b7deab6b.tar.gz
kernelci-pipeline-tarball             | 07/31/2023 11:33:28 PM UTC [INFO] set -e
kernelci-pipeline-tarball             | cd /home/kernelci/data/src/linux
kernelci-pipeline-tarball             | git archive --format=tar --prefix=linux-mainline-master-v6.5-rc4-1-g66f6b7deab6b/ HEAD | gzip > ../../output/linux-mainline-master-v6.5-rc4-1-g66f6b7deab6b.tar.gz
kernelci-pipeline-tarball             |
kernelci-pipeline-tarball             |
kernelci-pipeline-tarball             | 07/31/2023 11:33:55 PM UTC [INFO] Tarball created

@yurinnick yurinnick mentioned this pull request Aug 1, 2023
Open
@gctucker
Copy link
Contributor

gctucker commented Aug 1, 2023

Where is the Patchwork data coming from? In other words, would be sending a Node object with that data to the API?

gctucker
gctucker reviewed Aug 12, 2023
View reviewed changes
Copy link
Contributor

@gctucker gctucker left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

One issue there is the git describe, as it will be different every time the patches get applied. It's meant to be the Git revision from the upstream repository, in this case it won't refer to anything usable with the patches added as local commits.

This is the part we need to clarify I think. Initially, we could just keep the git describe and sha1 values as the base revision from the repository. Then a child node could be added with meta-data about the patchset and a new tarball with the patches applied for example. There are different ways to design this, it's important because of the .revision field which is part of the Node schema. So we can't keep everything in .data as reporting will be confusing with mismatching git revisions.

I'll create a GitHub issue to cover this part, then once we have at least one functional implementation we can revisit this PR and update it accordingly. Looking at how KCIDB is handling patchsets should also help there, like we discussed in the meeting last Wednesday.

@yurinnick
Copy link
Contributor Author

So should we extend .revision to include patchset hash? What should be patchset hash?
I think my proposal would be implement patch/patchset hashing based on patchwork hasher and CKI approach described here:

  • For patch hash will be sha256 (sha1?) hash of normalized patch content
  • for patchset hash will be sha256(sha1?) hash of all patches in order they were applied

This way we'll store unique pair of base commit revision and patchset hash in the Node.revision.

@gctucker
Copy link
Contributor

Yes I think the patches have to be part of the revision data and not just some extra arbitrary data. It would ideally need to be compatible with KCIDB, either by using the exact same schema and mechanism or by having a clear translation mechanism.

There are different ways to implement this, and I think it can be done incrementally. I'll post some comments on kernelci/kernelci-api#307

@yurinnick yurinnick closed this Oct 21, 2023
@yurinnick yurinnick deleted the patchwork-apply branch October 21, 2023 09:21
@yurinnick
Copy link
Contributor Author

Replace it with Patchset service implementation: #342

@pawiecz pawiecz mentioned this pull request Jan 12, 2024
Closed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@gctucker gctucker gctucker left review comments

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

[RFC] Patch building and Webhooks API
2 participants
@yurinnick @gctucker